Computer and computer system for controlling object manipulation in immersive virtual space

ABSTRACT

Action of a head around which an HMD main body is worn is related to manipulation of an object in an immersive three-dimensional virtual space. Information on a sight in a virtual space may be determined based on information on head inclination sensed with an inclination sensor; an image of the sight in the virtual space may be generated based on the sight information in order to display the sight image in the head mounted display; an object placed on a reference sight line in the virtual space may be identified, the reference sight line being determined in correspondence with a predetermined position in the sight image; and an object the identified object may be manipulated in response to the identification of the object in accordance with action of inclining in a predetermined direction the head around which the head mounted display is worn.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of InternationalApplication No. PCT/JP2016/057822, filed on Mar. 11, 2016, entitled“COMPUTER PROGRAM AND COMPUTER SYSTEM FOR CONTROLLING OBJECTMANIPULATION IN IMMERSIVE VIRTUAL SPACE”, which claims priority based onthe Article 8 of Patent Cooperation Treaty from prior Japanese PatentApplication No. 2015-052897, filed on Mar. 17, 2015, the entire contentsof which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a computer program and a computersystem for controlling object manipulation in an immersive virtualspace. The present disclosure relates more specifically to a computerprogram and a computer system that cause a computer to control objectmanipulation in such a way that an object placed in an immersive virtualspace can be manipulated through head mounted display (HMD)/headoperation using an HMD worn around a user's head and operation of acontroller held by the user.

BACKGROUND ART

There is a known HMD that is worn around a user's head and can presentthe user with an image in a virtual space with the aid of a display orany other device disposed in front of the user. In particular, the HMDdisclosed in Patent Document 1 can display a 360-degree panoramic imagein a three-dimensional virtual space. Such an HMD typically includes avariety of sensors (an acceleration sensor and an angular velocitysensor, for example) and measures data on the attitude of an HMD mainbody. In particular, the direction of the sight line of the eyes lookingat the panoramic image can be changed in accordance with information onthe angle of rotation of the head. That is, when the user who wears theHMD rotates his/her own head, the direction of the sight line of theeyes looking at the 360-degree panoramic image is changed accordingly,whereby the user's immersive sensation in the video world is enhancedfor improvement in entertainment quality (see paragraphs [0004], [0005],[0006], and [Abstract] of Patent Document 1).

Patent Document 1 further discloses that the user's gesture in the formof motion of the head is related in advance to operation (display screenswitching, for example) on the screen of the HMD, and when the user'sgesture is identified on the basis of the inclination of the head or theacceleration thereof, screen operation related to the gesture isperformed. Inconvenience in controller operation resulting from the factthat the user who wears the HMD is unable to see his/her own hands andtherearound is therefore solved (see paragraphs [0008] to [0009] inPatent Document 1).

CITATION LIST Patent Document

[Patent Document 1] Japanese Patent Laid-Open No. 2013-258614

The above disclosure in the Patent Document 1, however, merely describesuse of information on head action acquired with a sensor to identify thesight of the panoramic image or to identify the user's gesture. It isexpected in the future that a large number of various applications thatsupport an immersive virtual space using an HMD will be developed forwidespread use of an HMD.

In a game application of related art using no HMD, complicated operationof a controller is typically required in many cases, and there is aconcern about difficulty in operation of the controller due to the factthat the user who wears the HMD is unable to see his/her own hands andtherearound. In this regard, in an immersive virtual space based on anHMD, information on head action is expected to be applied to a varietyof scenes during the startup of an application in addition to theadvantageous effect described above.

As an example, assume an application in which an object placed in athree-dimensional virtual space is manipulated. In the followingdetailed description of the present disclosure, in particular, assume abuilding block game application that allows manipulation of buildingblock objects placed in an immersive three-dimensional virtual space,such as those shown in FIGS. 11 to 13. Action of the head around whichan HMD main body is worn is related to manipulation of moving a buildingblock object in a specific direction or any other manipulation, wherebyaction of the HMD allows manipulation of the building block object.

SUMMARY Technical Problem

An object of the present disclosure is to readily allow objectmanipulation in an immersive three-dimensional virtual space by relatingaction of a head around which an HMD main body is worn to specificmanipulation of an object placed in the three-dimensional virtual space.

Solution to Problem

A computer program according to the present disclosure that causes acomputer to control object manipulation in an immersive virtual spacecauses the computer to function as: a sight determination section thatdetermines information on a sight in the virtual space based oninformation on inclination sensed with an inclination sensor provided ina head mounted display connected to the computer; an image generationsection that generates an image of the sight in the virtual space basedon the sight information in order to display the sight image in the headmounted display; an object identification section that identifies anobject placed on a reference sight line in the virtual space, thereference sight line being determined in correspondence with apredetermined position in the sight image; and an object manipulationsection that manipulates the identified object in response to theidentification of the object in accordance with action of inclining thehead mounted display in a predetermined direction.

The computer program according to the present disclosure may cause thesight determination section to determine the sight information basedfurther on information on a position of the head mounted display that issensed with a position sensor connected to the computer and capable ofsensing the head mounted display.

Further, the computer program according to the present disclosure maycause the object manipulation section to manipulate the identifiedobject in cooperation with an external controller connectable to thecomputer. Moreover, the external controller may include a touch display,and the manipulation performed on the identified object is performed inaccordance with touch action including any of tapping, swiping, andholding performed on the touch display.

A computer system according to the present disclosure for objectmanipulation in an immersive virtual space includes a computer and ahead mounted display connected to the computer and including aninclination sensor, and the computer determines information on a sightin the virtual space based on information on inclination sensed with theinclination sensor, generates an image of the sight in the virtual spacebased on the sight information in order to display the sight image inthe head mounted display, identifies an object placed on a referencesight line in the virtual space, the reference sight line beingdetermined in correspondence with a predetermined position in the sightimage, and manipulates the identified object in response to theidentification of the object in accordance with action of inclining thehead mounted display in a predetermined direction.

According to the present disclosure, simple action of the head of a userwho wears an HMD readily allows object manipulation in an immersivethree-dimensional virtual space without complicated controller actionrequired in related art. Further, combining head action using the HMDwith controller action allows more complicated object manipulation. Thepresent disclosure can therefore provide a novel game operation aspectin a game application using an HMD.

The features and advantageous effects of the present disclosuredescribed above and other features and advantageous effects thereof willbe apparent from the following more specific description of examples ofthe present disclosure, the accompanying drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic view of an HMD system according to at least oneembodiment.

FIG. 2 shows an orthogonal coordinate system in an actual space definedwith respect to the center of the head of a user who wears the HMD shownin FIG. 1.

FIG. 3 is an outline view showing a plurality of sensing pointsvirtually provided on the HMD and sensed with a position trackingcamera.

FIG. 4 is a block diagram showing a functional configuration forimplementing object manipulation in a three-dimensional virtual spaceaccording to at least one embodiment.

FIG. 5 is a stereoscopic outline view diagrammatically showing anexample of the arrangement of a virtual camera and objects in thethree-dimensional virtual space according to at least one embodiment.

FIG. 6 is a stereoscopic outline view diagrammatically showing anexample of the arrangement of a virtual camera and objects in thethree-dimensional virtual space according to at least one embodiment.

FIG. 7A is a stereoscopic outline view diagrammatically showing anexample of manipulation of the virtual camera and the object in thethree-dimensional virtual space according to at least one embodiment.

FIG. 7B is a stereoscopic outline view diagrammatically showing anotherexample of manipulation of the virtual camera and the object in thethree-dimensional virtual space according to at least one embodiment.

FIG. 8 is a block diagram showing a functional configuration of anobject control section according to at least one embodiment.

FIG. 9 is an outline view showing an example of correspondence betweenobject manipulation and an action of a user according to at least oneembodiment.

FIG. 10 is a flowchart of processes for object manipulation in thethree-dimensional virtual space according to at least one embodiment.

FIG. 11 shows an example of a screen image relating to objectmanipulation in the three-dimensional virtual space according to atleast one embodiment.

FIG. 12 shows another example of the screen image relating to objectmanipulation in the three-dimensional virtual space according to atleast one embodiment.

FIG. 13 shows another example of the screen image relating to objectmanipulation in the three-dimensional virtual space according to atleast one embodiment.

DESCRIPTION OF EMBODIMENTS

A computer program that causes a computer to control object manipulationin an immersive virtual space and a computer system for objectmanipulation in an immersive virtual space according to at least oneembodiment will be described below with reference to the drawings. Inthe drawings, the same components are labeled with the same referencecharacters.

FIG. 1 is an overall schematic view of a head mounted display(hereinafter referred to as “HMD”) system 100 using an HMD according toat least one embodiment. The HMD system 100 includes an HMD main body110, a computer (control circuit section) 120, a position trackingcamera (position sensor) 130, and an external controller 140, as shownin FIG. 1.

The HMD 110 includes a display 112 and a sensor 114. The display 112 isa non-transmissive display device configured so as to completely cover auser's sight, and the user is thereby allowed to view only a screendisplayed in the display 112. Since the user who wears thenon-transmissive HMD 110 entirely loses sight of the outside world,there is achieved a display aspect in which the user is completelyimmersed in a virtual space displayed by an application executed in thecontrol circuit section 120.

The sensor 114 provided in the HMD 110 is fixed to a portion in thevicinity of the display 112. The sensor 114 includes a terrestrialmagnetism sensor, an acceleration sensor, and/or an inclination (angularvelocity, gyro) sensor and can sense a variety of types of motion of theHMD 110 (display 112) worn around the user's head by using at least oneof the sensors. The angular velocity sensor, in particular, can senseangular velocities of the HMD 110 around the three axes thereof in atime course in accordance with motion of the HMD 110 and determine atemporal change in angle (inclination) around each of the axes, as shownin FIG. 2.

Angular information data that can be sensed with the inclination sensorwill be described with reference to FIG. 2. XYZ coordinates are definedwith respect to the head of the user who wears the HMD, as shown in FIG.2. The Y axis extends in the vertical direction in which the user standsup. The Z axis is orthogonal to the Y axis and extends in the directionin which the center of the display 112 and the user are connected toeach other. The X axis is an axis extending in the direction orthogonalto the Y axis and the Z axis. The inclination sensor senses an anglearound each of the axes (that is, inclination determined by the yawangle representing rotation about the Y axis, the pitch anglerepresenting rotation about the X axis, and the roll angle representingrotation about the Z axis), and a motion sensing section 210 determinesangular (inclination) information data as information on the sight onthe basis of the time-course change.

Referring back to FIG. 1, the computer 120 provided in the HMD system100 functions as the control circuit section 120 that causes the userwho wears the HMD to be immersed in a three-dimensional virtual spaceand further causes the user to perform action based on thethree-dimensional virtual space. The control circuit section 120 may beconfigured as hardware separate from the HMD 110, as shown in FIG. 1.The hardware can be a computer, such as a personal computer or a servercomputer that operates over a network. That is, the hardware can be anarbitrary computer including a CPU, a main storage, an auxiliarystorage, a transceiver, a display section, and an input sectionconnected to each other via a bus.

The control circuit section 120 may instead be accommodated as an objectmanipulator in the HMD 110. In this case, the control circuit section120 can implement the entirety or only some of the functions of theobject manipulator. In the case where only some of the functions areimplemented, the remaining functions may be implemented in the HMD 110or a server computer (not shown) that operates over a network.

The position tracking camera (position sensor) 130 provided in the HMDsystem 100 is so connected to the control circuit section 120 that theycan communicate with each other and has the function of tracking theposition of the HMD 110. The position tracking camera 130 is achieved byusing an infrared sensor and a plurality of optical cameras. The HMDsystem 100, which includes the position tracking camera 130 and sensesthe position of the HMD around the user's head, can accurately relatethe virtual space positions of the virtual camera/immersed user in thethree-dimensional virtual space to each other and identify the virtualspace positions.

More specifically, the position tracking camera 130 is virtuallyprovided on the HMD 110, as shown in FIG. 3 by way of example, andsenses, in a time course in correspondence with the user's motion,actual space positions of a plurality of sensing points where infraredlight is sensed. A temporal change in the angle around each of the axesaccording to the motion of the HMD 110 can then be determined on thebasis of the time-course change in the actual space positions sensed bythe position tracking camera 130.

Referring back to FIG. 1, the HMD system 100 includes the externalcontroller 140. The external controller 140 is a typical user terminaland can be a smartphone shown in FIG. 1 but is not limited thereto. Theexternal controller 140 can instead, for example, be a portable deviceterminal including a touch display, such as a PDA, a tablet computer, agame console, or a notebook PC. That is, the external controller 140 canbe an arbitrary portable device terminal including a CPU, a mainstorage, an auxiliary storage, a transceiver, a display section, and aninput section connected to each other via a bus. The user can perform avariety of types of touch action including tapping, swiping, and holdingon the touch display of the external controller 140.

The block diagram of FIG. 4 shows the configuration of primary functionsof the control circuit section 120 and other components therearound toimplement object manipulation in the three-dimensional virtual spaceaccording to at least one embodiment. The control circuit section 120primarily receives inputs from the sensors 114/130 and the externalcontroller 140, processes the inputs, and outputs results of theprocessing to the display 112. The control circuit section 120 primarilyincludes the motion sensing section 210, a sight determination section220, a sight image generation section 230, and an object control section240 and interacts with a variety of tables in a spatial informationstorage section 250, an object information storage section 260, avirtual camera information storage section 270, and other sections toprocess a variety of pieces of information.

The motion sensing section 210 measures data on motion of the HMD 110worn around the user's head on the basis of motion information inputtedfrom the sensors 114/130. In at least one embodiment, in particular,angular information sensed in a time course with the inclination sensor114 and position information sensed in a time course with the positiontracking camera 130 are determined.

The sight determination section 220 determines sight information of thevirtual cameras in the three-dimensional virtual space on the basis ofthree-dimensional virtual space information stored in the spatialinformation storage section 250, angular information sensed with theinclination sensor 114, and sensed information in the direction of thefield of view of the virtual cameras based on the position informationsensed with the position sensor 130. The sight image generation section230 can then generate a sight image of part of a 360-degree panoramicimage on the basis of the determined sight information. As the sightimage, two two-dimensional images for the right and left eyes aregenerated and superimposed on each other in the HMD, and thesuperimposed image displayed in the HMD is presented to the user in theform of a three-dimensional image, as shown in FIGS. 11 to 13.

Referring back to FIG. 4, the object control section 240 identifies anobject to be manipulated on the basis of object information of an objectin the three-dimensional virtual space that is stored in the objectinformation storage section 260, information from the sensors 114/130,and the user's instructions from the external controller 140. The objectcontrol section 240 receives an instruction of predetermined useroperation to be performed on the identified object to be manipulated,adjusts virtual camera information stored in the virtual camerainformation storage section 270 and manipulates the object to bemanipulated, and outputs a result of the manipulation to the display 112of the HMD. Specific processes of the object manipulation will bedescribed in association with FIG. 5 and the following figures.

In FIG. 4, elements drawn as functional blocks that carry out a varietyof processes can be configured by a CPU, a memory, and other integratedcircuits in a hardware sense and can be achieved by a variety ofprograms loaded in the memory in a software sense. A person skilled inthe art therefore understands that these functional blocks can beachieved by any of the hardware, the software, and a combination thereof(the same holds true for FIG. 8, which will be described later).

The process of identifying an object to be manipulated and the processof manipulating the object to be manipulated that are carried out by thesight determination section 220, the sight image generation section 230,and the object control section 240 shown in FIG. 4 will be described indetail with reference to FIGS. 5 to 10. FIG. 5 is a stereoscopic viewdiagrammatically showing an example of the arrangement of a virtualcamera 1, cylindrical objects O1 and O2, and a cubic object O3 in animmersive three-dimensional virtual space 2 according to at least oneembodiment. The sight captured with the virtual camera 1 is displayed asthe sight image in the display of an immersed user. Since the head ofthe user who wears the HMD inclines downward, the virtual camera 1 isalso oriented to a point below a horizontal line, as shown in FIG. 5.The cylindrical object O1 is placed at the front end of the (arrowed)center line of the sight. Further, the cylindrical object O2 is placedon the right in the sight, and the cubic object O3 is placed on the leftin the sight.

FIG. 6 shows the arrangement in the diagrammatic view of FIG. 5 indetail. As shown in FIG. 6, in a virtual space in which a horizontalplane is defined by the XZ coordinates and the vertical direction isdefined by the Z axis, the virtual camera 1 is placed in a position(X_(cam), Y_(cam), Z_(cam)). The coordinates of the position aredetermined on the basis of the position information sensed with theposition tracking camera. The field of view of the virtual camera 1 hasa predetermined range, and the sight is defined on the basis of therange of the field of view. A sight image is generated on the basis ofthe defined sight. A sight reference sight line L_(std) and sightboundary lines L_(bd1) and L_(bd2) with respect to the reference sightline are defined. The reference line L_(std) is so defined as tocorrespond to a predetermined position (a central point, for example) inthe sight image.

The objects O1 to O3 are placed in a position (X_(O1), 0, Z_(O1)), aposition (X_(O2), 0, Z_(O2)), and a position (X_(O3), 0, Z_(O3)),respectively, in the XZ plane by way of example. Since an object, ofcourse, has a fixed size, the coordinate position of the object may bedefined in the form of a coordinate range. In the relationship in theexample in FIG. 6 between the position and orientation of the virtualcamera and the positions where the objects are placed, the object O1 isplaced on the reference sight line L_(std). As a result, the object O1is identified as an object to be manipulated.

FIGS. 7A and 7B each show an outline of manipulation performed on theobject O1 identified in FIG. 6. FIG. 7A diagrammatically shows anexample of manipulation of the virtual camera and the object in thethree-dimensional virtual space according to at least one embodiment.When the object O1 to be manipulated is identified, the position of thevirtual camera 1 is fixed, as shown in FIG. 7A. In this state, when theuser performs HMD action of further inclining the head, objectmanipulation related in advance to the action is performed on the objectO1. In the description, the user performs HMD inclining action of movingthe head in the vertical direction, and the object O1 is moved also inthe three-dimensional virtual space in accordance with the distance ofthe head movement in the Y-axis direction to a position havingcoordinates (X_(O1), Y_(O1), Z_(O1)). In this process, the orientationof the virtual camera is adjusted in terms of angle also in the Y-axisdirection in such a way that a reference sight line L_(OP′) always givesa sight reference position (for example, the central position in thesight image is maintained).

In FIG. 7A, an object O1′ having moved in the Y-axis direction isfurther rotated. The object rotation can also be related topredetermined HMD inclining action (action of inclining HMD inhorizontal direction, for example), as in the object movement. Theobject rotation can instead be related to predetermined action performedon the external controller 140 (swiping on touch display, for example).

FIG. 7B diagrammatically shows another example of manipulation of thevirtual camera and the object in the three-dimensional virtual spaceaccording to at least one embodiment. In the example in FIG. 7A, afterthe object O1 to be manipulated is identified, the position of thevirtual camera 1 is fixed, and the orientation of the camera is thenadjusted. In contrast, in the example in FIG. 7B, the position of thevirtual camera is not fixed, but the orientation thereof is fixed. Thatis, when the object O1 is moved in the Y-axis direction to the positionhaving the coordinates (X_(O1), Y_(O1), Z_(O1)), the position of thevirtual camera 1 is also moved in the Y direction to the position havingcoordinates (X_(cam), Y_(cam+)Y_(O1), Z_(cam)).

The control of the position and orientation of a virtual camera is notlimited to the process example in FIG. 7A or 7B. That is, a personskilled in the art should understand that there can be employed anyprocess aspect in which the position and/or orientation of a virtualcamera is so adjusted that the reference sight line L_(OP′) always givesthe sight reference position (for example, the central position in thesight image is maintained).

Processes relating to the object manipulation in the three-dimensionalvirtual space having been described above with reference to FIGS. 6, 7A,and 7B will be described in more detail with reference to FIGS. 8 to 10.FIG. 8 is a detailed functional block diagram showing blocks that formthe object control section 240 having been described with reference toFIG. 4. The object control section 240 includes an object identificationsection 300 for identifying an object to be manipulated and an objectmanipulation section 400 for manipulating the identified object to bemanipulated, as shown in FIG. 8.

The object identification section 300 identifies an object placed on thereference sight line shown in FIG. 6 in the three-dimensional virtualspace on the basis of the user's HMD inclining action. To this end, theobject identification section 300 includes a reference sight linecalculation section 320, which calculates a reference sight line that ispresent in the three-dimensional virtual space and related to areference point in a sight image, an object evaluation section 340,which evaluates whether an object is placed on the reference sight line,and a manipulated object selection section 360, which selects the objectas an object to be manipulated if the object is placed on the referencesight line.

On the other hand, the object manipulation section 400 responds to theidentification of an object to be manipulated performed by the objectidentification section 300 and performs on the object to be manipulatedmanipulation according to the user's action of inclining the head aroundwhich the HMD is worn in a predetermined direction. To this end, theobject manipulation section 400 includes an external controller actionevaluation section 420, which evaluates whether the external controllerhas received the user's touch action, an inclining action evaluationsection 440, which evaluates the user's HMD inclining action(inclination direction), a virtual camera adjustment section 450, whichadjusts the position or direction of a virtual camera when objectmanipulation is performed, an object manipulation identification section460, which identifies object manipulation performed on an object to bemanipulated in accordance with an object manipulation table 470, and anobject manipulation performing section 480, which performs the objectmanipulation to generate a sight image.

The object manipulation identification using the object manipulationtable 470 will be described with reference to FIG. 9. FIG. 9 is aschematic view showing an example of correspondence between the user'saction and object manipulation, on the basis of which the objectmanipulation table 470 is created. When the manipulated object selectionsection 360 selects an object to be manipulated, and the usercontinuously performs “tapping” on the external controller, the objectto be manipulated can be “caught” in the three-dimensional virtualspace. The “tapping” is presented only by way of example, and there maybe used any other HMD action or input action that is performed by theuser and can be uniquely related to object manipulation in a state inwhich an object to be manipulated is selected.

Object manipulation after the “catching” described above can bedetermined in accordance with the object manipulation table 470. Theuser's action includes not only the HMD inclining action (column) butalso touch action performed on the external controller (row), andcombinations of these types of user's action allow identification ofobject manipulation. For example, in a case where the HMD action is“upward or downward inclination” and the user further performs “tapping”on the touch display as external controller action, the overall actionis taken as an “object releasing” action instruction. Similarly, in acase where the HMD action is “upward or downward inclination” and theuser further performs “swiping” in cooperation with the externalcontroller, the overall action is taken by the object manipulationidentification section 460 as an “object rotating” action instruction;when the user further performs “holding” on the external controller, theoverall action is taken as an “object upward or downward movement”action instruction; and when the user further performs nothing asexternal controller action, the overall action is taken as an “objectfrontward or rearward movement” action instruction. The objectmanipulation performing section 480 then performs each of the actioninstructions.

The HMD action is not limited to “upward or downward inclination” andmay be any inclination action that is uniquely identifiable. Further,only one of HMD inclining action and external controller action may beaccepted as the user's action and related to object manipulation.However, in a case where it is desired to provide a large number ofobject manipulation types, it is preferable to use combinations of HMDinclining action and external controller action. The user's HMDinclining action, which can be performed only by causing the user whowears the HMD to move his/her own head, is easy for the user. Further,it can be said that touch action performed on the external controller isalso easy for the user because the user only needs to perform “tapping,”“swiping,” or “holding” in any position on the touch display.

The procedure of the processes for object manipulation in thethree-dimensional virtual space according to at least one embodimentwill next be described in detail with reference to FIG. 10. The objectmanipulation processes are achieved by interaction among the externalcontroller 140, the position tracking camera (position sensor) 130, theHMD 110, and the computer (control circuit section) 120.

In the object manipulation processes, the motion sensing section 210senses a variety of pieces of information on the HMD. That is, in stepS130-1, the position sensor connected to the computer 120 and capable ofsensing the HMD senses the position of the HMD 110. In step S100-1, theinclination sensor provided in the HMD connected to the computer 120senses the inclination of the HMD 110. In step S120-1, the motionsensing section 210 determines the position information and theinclination information, and the sight determination section 220determines the sight information in the three-dimensional virtual spaceon the basis of the position information and/or the inclinationinformation on the HMD 110 described above.

After the sight information is determined, in the subsequent stepS120-2, the sight image generation section 230 generates a sight imageto be displayed in the HMD on the basis of the sight informationdescribed above. Further, in step S120-3, the reference sight linecalculation section 320 determines the reference sight line L_(std). Thereference sight line L_(std) is determined in correspondence with apredetermined position in the sight image. The predetermined position ispreferably but not necessarily the central point of the sight image andmay be any settable position in the sight image. Further, the referencepoint is preferably so displayed in the form of a mark of some kind(palm icon, for example) as to be superimposed on the sight imagedescribed above.

Thereafter, in step S100-2, the sight image is displayed in the HMD. Aplurality of objects each of which is a candidate of an object to bemanipulated later are displayed in the sight image. In the state inwhich the sight image is displayed in the HMD, the user who wears theHMD performs head inclining action to perform positioning in such a waythat at least one of the plurality of objects placed in thethree-dimensional virtual space corresponds to the predeterminedposition in the sight image.

In step S120-4, the object evaluation section 340 evaluates whether atleast one object has been positioned in the predetermined positiondescribed above in the sight image; that is, whether an object has beenplaced on the reference sight line in the three-dimensional space. Inthe case where an object has been placed on the reference sight line, instep S120-5, the manipulated object selection section 360 selects theobject as an object to be manipulated. If a plurality of objects placedon the reference sight line are present, the manipulated objectselection section 360 may select one object closest to the position ofthe virtual camera in the three-dimensional virtual space; that is, thepoint of view.

In step S140-1, it is evaluated whether the user has performed touchaction (“tapping,” in particular) on the external controller 140 afterthe selection of the object in step S120-5. In the case where the userhas performed touch action, in step S120-6 the object identificationsection 300 responds to the touch action, and identification of anobject to be manipulated is completed. In this state, the sight image isdisplayed in the HMD in an aspect in which the user has “caught” theobject to be manipulated. On the other hand, in the case where the userhas performed no touch action, the control returns to step S130-1 andstep S100-1 in the initial stage, and the information on the positionand inclination of the HMD is continuously sensed.

In the state in which the user has caught the object to be manipulated,the user further performs external controller action (step S140-2) andHMD inclining action (step S100-3). In response to the action, theexternal controller action evaluation section 420 and the incliningaction/direction evaluation section 440 perform action evaluation, andin step S120-7, the object manipulation identification section 460identifies corresponding manipulation performed on the object to bemanipulated. The corresponding manipulation has been described withreference to FIG. 9. Unique object manipulation is determined on thebasis a combination of HMD inclining action; for example, in thevertical direction (HMD upward or downward inclination action) andexternal controller action.

In step S120-8 and the following steps, actual manipulation is performedon the object to be manipulated, and a result of the manipulation isdisplayed in the HMD. That is, in step S120-8, the virtual cameraadjustment section 450 adjusts the position and direction of the virtualcamera shown in FIGS. 7A and 7B. In the subsequent step S120-9, theobject manipulation performing section 480 performs specificmanipulation on the object to be manipulated, as shown in FIGS. 7A and7B again. In step S120-10, a series of images resulting from themanipulation are generated, and in step S100-4, the images resultingfrom the manipulation are displayed in the HMD in a time course.

Examples of Screen Display

On the basis of the above description, FIGS. 11 to 13 show screenexamples displayed in the HMD that are implemented on the basis of atleast one embodiment and relate to object manipulation in thethree-dimensional virtual space. In the examples, assume a buildingblock game application that allows manipulation of building blockobjects placed in an immersive three-dimensional virtual space. Morespecifically, assume a building block game application in which HMD headaction is related to manipulation of moving a building block object in aspecific direction or any other manipulation so that the user canmanipulate the building block object by moving the head around which theHMD is worn. In FIGS. 11 to 13, two images for the right and left eyesare displayed, and when the images are displayed in the HMD, they aresuperimposed on each other as if the user sees a three-dimensionalimage, as described above.

In FIG. 11, a large number of building block objects are placed on aplane in the three-dimensional virtual space. Further, a mark having apalm shape is displayed at the center of a sight image. The user thenperforms HMD inclining action for positioning in such a way that themark overlaps with any of the building block objects to advance thegame. The user selects a single building block object in the positioningprocess and then combines HMD inclining action with external controlleraction to manipulate the building block object in accordance with anobject manipulation rule, such as those shown in FIGS. 7A and 7B. FIG.12 shows a state in which the selected, caught building block object ismoved to a predetermined position. FIG. 13 shows a state in which asingle building block object is lifted upward by HMD inclining action inthe vertical direction and the lifted building block object is to bestacked on another building block object placed in a position adjacentto the lifted building block object.

As described above, according to the present disclosure, simple actionof the head of a user who wears an HMD allows object manipulation in animmersive three-dimensional virtual space. Further, combining headaction using the HMD with external controller action allows morecomplicated object manipulation. Based on the features described above,the present disclosure can provide a novel game operation aspect in agame application using an HMD.

At least one embodiment has been described above, but the presentdisclosure is not limited to the at least one embodiment describedabove. A person skilled in the art will understand that a variety ofmodification can be made to the at least one embodiment withoutdeparting from the spirit and scope of the present disclosure set forthin the claims.

DESCRIPTION OF SYMBOLS

-   1 Virtual camera-   2 Immersive three-dimensional virtual space-   100 HMD system-   110 HMD-   112 Display-   114 Inclination sensor (gyro sensor)-   120 Computer (control circuit section)-   130 Position sensor (position tracking camera)-   140 External controller-   210 Motion sensing section-   220 Sight determination section-   230 Sight image generation section-   240 Object control section-   250 Spatial information storage section-   260 Object information storage section-   270 Virtual camera information storage section-   300 Object identification section-   320 Reference sight line calculation section-   340 Object evaluation section-   360 Manipulated object selection section-   400 Object manipulation section-   420 External controller action evaluation section-   440 Inclining action evaluation section-   450 Virtual camera adjustment section-   460 Object manipulation identification section-   470 Object manipulation table-   480 Object manipulation performing section

What is claimed is:
 1. A computer comprising a CPU and a memory having acomputer program stored therein wherein the computer program, whenexecuted by the CPU causes the CPU to perform operations comprising:determining information on a sight in a three-dimensional virtual spacebased on three-dimensional virtual space information stored in thememory and information on inclination sensed with an inclination sensorprovided in a head mounted display connected to the computer; generatingan image of the sight in the three-dimensional virtual space based onthe sight information in order to display the sight image in the headmounted display; identifying an object in the generated sight imageplaced on a reference sight line in the generated sight image in thethree-dimensional virtual space, the reference sight line beingdetermined in correspondence with a predetermined position in thegenerated sight image; and manipulating the identified object in thethree-dimensional virtual space in accordance with action of incliningthe head mounted display in a predetermined direction.
 2. The computeraccording to claim 1, wherein the determining sight information basedfurther on information on a position of the head mounted display that issensed with a position sensor connected to the computer and capable ofsensing the head mounted display.
 3. The computer according to claim 1,wherein in a case where a plurality of objects are placed on thereference sight line, the identifying an object in the generated sightimage comprises selecting an object closest to a point of view in thethree-dimensional virtual space.
 4. The computer according to claim 1,wherein the predetermined position is set at a central point of thesight image.
 5. The computer according to claim 1, wherein thepredetermined direction is a vertical direction.
 6. The computeraccording to claim 1, wherein the manipulating the identified object inthe three-dimensional virtual space comprises manipulating theidentified object in cooperation with an external controller connectableto the computer.
 7. The computer according to claim 6, wherein theexternal controller includes a touch display, and the manipulationperformed on the identified object in cooperation with the externalcontroller is performed in accordance with touch action including any oftapping, swiping, and holding performed on the touch display.
 8. Acomputer system comprising a computer and a head mounted displayconnected to the computer and including an inclination sensor, whereinthe computer is configured to perform operations comprising: determininginformation on a sight in a three-dimensional virtual space based onstored three-dimensional virtual space information and information oninclination sensed with the inclination sensor, generating an image ofthe sight in the three-dimensional virtual space based on the sightinformation in order to display the sight image in the head mounteddisplay, identifying an object in the generated sight image placed on areference sight line in the generated sight image in thethree-dimensional virtual space, the reference sight line beingdetermined in correspondence with a predetermined position in thegenerated sight image, and manipulating the identified object in thethree-dimensional virtual space in accordance with action of incliningthe head mounted display in a predetermined direction.
 9. The computersystem according to claim 8, further comprising a position sensorconnected to the computer and capable of sensing the head mounteddisplay, wherein the sight information is determined based further oninformation on a position of the head mounted display sensed with theposition sensor.
 10. The computer system according to claim 8, furthercomprising an external controller connectable to the computer, whereinthe manipulation performed on the identified object is performed inaccordance with touch action performed on a touch display provided inthe external controller.
 11. A non-transitory computer-readable mediumhaving a computer program stored therein, the computer program when readand executed by a processor causing the processor to perform operationscomprising: determining information on a sight in a three-dimensionalvirtual space based on stored three-dimensional virtual spaceinformation and information on sensed inclination of a head mounteddisplay; generating an image of the sight in the three-dimensionalvirtual space based on the sight information in order to display thesight image in the head mounted display; identifying an object in thegenerated sight image placed on a reference sight line in the generatedsight image in the three-dimensional virtual space, the reference sightline being determined in correspondence with a predetermined position inthe generated sight image; and manipulating the identified object in thethree-dimensional virtual space in accordance with action of incliningthe head mounted display in a predetermined direction.